Reflections on Software Failure Analysis

Failure studies are important in revealing the root causes, behaviors, and life cycle of defects in software systems. These studies either focus on understanding the characteristics of defects in specific classes of systems or the characteristics of a specific type of defect in the systems it manifests in. Failure studies have influenced various software engineering research directions, especially in the area of software evolution, defect detection, and program repair. In this paper, we reflect on the conduct of failure studies in software engineering. We reviewed a sample of 52 failure study papers. We identified several recurring problems in these studies, some of which hinder the ability of the engineering community to trust or replicate the results. Based on our findings, we suggest future research directions, including identifying and analyzing failure causal chains, standardizing the conduct of failure studies, and tool support for faster defect analysis

Reflections on Software Failure Analysis

Failure studies are important in revealing the root causes, behaviors, and life cycle of defects in software systems. These studies either focus on understanding the characteristics of defects in specific classes of systems or the characteristics of a specific type of defect in the systems it manifests in. Failure studies have influenced various software engineering research directions, especially in the area of software evolution, defect detection, and program repair. In this paper, we reflect on the conduct of failure studies in software engineering. We reviewed a sample of 52 failure study papers. We identified several recurring problems in these studies, some of which hinder the ability of the engineering community to trust or replicate the results. Based on our findings, we suggest future research directions, including identifying and analyzing failure causal chains, standardizing the conduct of failure studies, and tool support for faster defect analysis.Comment: 6 pages, 4 figures To be published in: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '22

Systematically Detecting Packet Validation Vulnerabilities in Embedded Network Stacks

Embedded Network Stacks (ENS) enable low-resource devices to communicate with the outside world, facilitating the development of the Internet of Things and Cyber-Physical Systems. Some defects in ENS are thus high-severity cybersecurity vulnerabilities: they are remotely triggerable and can impact the physical world. While prior research has shed light on the characteristics of defects in many classes of software systems, no study has described the properties of ENS defects nor identified a systematic technique to expose them. The most common automated approach to detecting ENS defects is feedback-driven randomized dynamic analysis ("fuzzing"), a costly and unpredictable technique. This paper provides the first systematic characterization of cybersecurity vulnerabilities in ENS. We analyzed 61 vulnerabilities across 6 open-source ENS. Most of these ENS defects are concentrated in the transport and network layers of the network stack, require reaching different states in the network protocol, and can be triggered by only 1-2 modifications to a single packet. We therefore propose a novel systematic testing framework that focuses on the transport and network layers, uses seeds that cover a network protocol's states, and systematically modifies packet fields. We evaluated this framework on 4 ENS and replicated 12 of the 14 reported IP/TCP/UDP vulnerabilities. On recent versions of these ENSs, it discovered 7 novel defects (6 assigned CVES) during a bounded systematic test that covered all protocol states and made up to 3 modifications per packet. We found defects in 3 of the 4 ENS we tested that had not been found by prior fuzzing research. Our results suggest that fuzzing should be deferred until after systematic testing is employed.Comment: 12 pages, 3 figures, to be published in the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE 2023

Preventing Supply Chain Vulnerabilities in Java with a Fine-Grained Permission Manager

Integrating third-party packages accelerates modern software engineering, but introduces the risk of software supply chain vulnerabilities. Vulnerabilities in applications' dependencies are being exploited worldwide. Often, these exploits leverage features that are present in a package, yet unneeded by an application. Unfortunately, the current generation of permission managers, such as SELinux, Docker containers, and the Java Security Manager, are too coarse-grained to usefully support engineers and operators in mitigating these vulnerabilities. Current approaches offer permissions only at the application's granularity, lumping legitimate operations made by safe packages with illegitimate operations made by exploited packages. This strategy does not reflect modern engineering practice. we need a permission manager capable of distinguishing between actions taken by different packages in an application's supply chain. In this paper, we describe Next-JSM, the first fine-grained ("supply chain aware") permission manager for Java applications. Next-JSM supports permission management at package-level granularity. Next-JSM faces three key challenges: operating on existing JVMs and without access to application or package source code, minimizing performance overhead in applications with many packages, and helping operators manage finer-grained permissions. We show that these challenges can be addressed through bytecode rewriting; appropriate data structures and algorithms; and an expressive permission notation plus automated tooling to establish default permission. In our evaluation, we report that Next-JSM mitigates 11 of the 12 package vulnerabilities we evaluated and incurs an average 2.72% overhead on the Dacapobench benchmark. Qualitatively, we argue that Next-JSM addresses the shortcomings of the (recently deprecated) Java Security Manager (JSM).Comment: 15 pages, 5 figures, 5 table